Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mass decoding non dynamic #6630

Merged
merged 136 commits into from
Oct 25, 2024
Merged

Mass decoding non dynamic #6630

merged 136 commits into from
Oct 25, 2024

Conversation

0xBoxer
Copy link
Collaborator

@0xBoxer 0xBoxer commented Aug 29, 2024

scaleable evm decode function usage

@0xBoxer 0xBoxer added WIP work in progress dune team created by dune team labels Aug 30, 2024
@Hosuke
Copy link
Collaborator

Hosuke commented Oct 15, 2024

We need to merge #6929 first to get the correct dbt test result.
(Dependency across dbt_subprojects.)

@jeff-dude
Copy link
Member

(new evms.contracts is available, retrying)

@jeff-dude jeff-dude removed the question Further information is requested label Oct 15, 2024
@jeff-dude
Copy link
Member

summarizing where we stand:

  • uniswap v2 & v3 both work, with ethereum and arbitrum chains enabled in this PR (more chains to follow this PR)
  • removed fork mappings static files
  • leverage evms.contracts to obtain project names to assign in dex
  • hardcode version numbers to match contract version numbers (sometimes forks can mismatch versions, not much we can do programatically afaik)

open question:

  • do we still need inner join to creation_traces in each uniswap macro? or does the downstream join to transfers do the same check? (for @0xBoxer)

our main test goal is to match existing prod dex.trades swaps for these projects per chain. while doing this test, we have output a few categories to check:

  • New in new_data -- 8,213,763 new rows!
  • Sold/Bought Mismatch -- 2,736,727 this is due to a bug in prod dex.trades for one specific project (kyber) flipping bought/sold, no action needed here
  • Token Bought Address Mismatch -- 214 this is where we need help on why they differ, seems like aggregator actions on a few specific tx hashes we checked 🤔
  • Token Sold Address Mismatch -- 22 likely same as bought above, looking for answer
  • Token Bought Amount Raw Mismatch -- 5 likely due to same issue as previous two?

request for @0xBoxer to help with the final three categories listed there (only ~240 rows total), hopefully same pattern to solve.

query to obtain above categories:

WITH
  new_data AS (
    SELECT
      tx_hash,
      evt_index,
      token_bought_amount_raw,
      token_sold_amount_raw,
      token_bought_address,
      token_sold_address,
      project,
      version,
      amount_usd,
      block_date,
      blockchain,
      project_contract_address
    FROM
      test_schema.git_dunesql_dd52f88_dex_automated_trades
  ),
  prod_data AS (
    SELECT
      tx_hash,
      evt_index,
      token_bought_amount_raw,
      token_sold_amount_raw,
      token_bought_address,
      token_sold_address,
      project,
      version,
      amount_usd,
      block_date,
      blockchain,
      project_contract_address
    FROM
      dex.trades
  ),
  final_results AS (
    SELECT
      CASE
        WHEN p.tx_hash IS NULL THEN 'New in new_data'
        WHEN n.token_bought_amount_raw = p.token_sold_amount_raw
        AND n.token_sold_amount_raw = p.token_bought_amount_raw
        AND n.token_bought_address = p.token_sold_address
        AND n.token_sold_address = p.token_bought_address THEN 'Sold/Bought Mismatch'
        WHEN n.token_bought_address != p.token_bought_address THEN 'Token Bought Address Mismatch'
        WHEN n.token_sold_address != p.token_sold_address THEN 'Token Sold Address Mismatch'
        WHEN n.token_bought_amount_raw != p.token_bought_amount_raw THEN 'Token Bought Amount Raw Mismatch'
        WHEN n.token_sold_amount_raw != p.token_sold_amount_raw THEN 'Token Sold Amount Raw Mismatch'
        WHEN ABS(n.amount_usd - p.amount_usd) / NULLIF(p.amount_usd, 0) > 0.10 THEN 'Amount USD Mismatch > 10%'
        ELSE 'Match'
      END AS comparison_result,
      COALESCE(n.block_date, p.block_date) AS block_date,
      COALESCE(n.blockchain, p.blockchain) AS blockchain,
      COALESCE(n.project, p.project) AS project,
      COALESCE(n.version, p.version) AS version,
      n.tx_hash,
      n.evt_index,
      n.token_bought_address AS new_token_bought_address,
      p.token_bought_address AS prod_token_bought_address,
      n.token_sold_address AS new_token_sold_address,
      p.token_sold_address AS prod_token_sold_address,
      n.token_bought_amount_raw AS new_token_bought_amount_raw,
      p.token_bought_amount_raw AS prod_token_bought_amount_raw,
      n.token_sold_amount_raw AS new_token_sold_amount_raw,
      p.token_sold_amount_raw AS prod_token_sold_amount_raw,
      n.amount_usd AS new_amount_usd,
      p.amount_usd AS prod_amount_usd,
      CASE
        WHEN n.token_bought_address = p.token_bought_address THEN 'Match'
        ELSE 'Mismatch'
      END AS token_bought_address_comparison,
      CASE
        WHEN n.token_sold_address = p.token_sold_address THEN 'Match'
        ELSE 'Mismatch'
      END AS token_sold_address_comparison,
      CASE
        WHEN n.token_bought_amount_raw = p.token_bought_amount_raw THEN 'Match'
        ELSE 'Mismatch'
      END AS token_bought_amount_raw_comparison,
      CASE
        WHEN n.token_sold_amount_raw = p.token_sold_amount_raw THEN 'Match'
        ELSE 'Mismatch'
      END AS token_sold_amount_raw_comparison,
      CASE
        WHEN ABS(n.amount_usd - p.amount_usd) / NULLIF(p.amount_usd, 0) <= 0.10 THEN 'Match'
        ELSE 'Mismatch'
      END AS amount_usd_comparison,
      n.project_contract_address
    FROM
      new_data n
      LEFT JOIN prod_data p ON n.tx_hash = p.tx_hash
      AND n.evt_index = p.evt_index
      and n.blockchain = p.blockchain
      and n.block_date = p.block_date
    WHERE
      (
        p.tx_hash IS NULL
        OR n.token_bought_address != p.token_bought_address
        OR n.token_sold_address != p.token_sold_address
        OR n.token_bought_amount_raw != p.token_bought_amount_raw
        OR n.token_sold_amount_raw != p.token_sold_amount_raw
        OR ABS(n.amount_usd - p.amount_usd) / NULLIF(p.amount_usd, 0) > 0.10
      )
  )
SELECT
  comparison_result,
  count(1)
FROM
  final_results
group by
  comparison_result

@0xBoxer
Copy link
Collaborator Author

0xBoxer commented Oct 23, 2024

Due to a dbt version change, you need to merge main into your branch before the Github action runner runs successfully again.

@jeff-dude jeff-dude added ready-for-merging and removed in review Assignee is currently reviewing the PR labels Oct 25, 2024
@jeff-dude jeff-dude merged commit 1322745 into main Oct 25, 2024
1 of 2 checks passed
@jeff-dude jeff-dude deleted the mass-decoding-non-dynamic branch October 25, 2024 18:25
@github-actions github-actions bot locked and limited conversation to collaborators Oct 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
dbt: dex covers the DEX dbt subproject dune team created by dune team ready-for-merging
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants